[configure] Configure Kubelet Log Level Verbosity#147
Conversation
WalkthroughAdded new troubleshooting documentation explaining how to increase kubelet log verbosity for diagnosing node-level issues. The guide covers supported verbosity ranges (0–10), configuration procedures for kubeadm-based clusters, systemd drop-in methods, immutable-OS nodes, and verification steps via kubectl debug and journalctl. Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~5 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/en/solutions/Configure_Kubelet_Log_Level_Verbosity.md`:
- Around line 126-128: The loop uses node names (variable n and ssh "$n") which
may not be DNS-resolvable; change the selection to use each node's InternalIP
and keep the node name for filenames: iterate over pairs (node name and
InternalIP) by replacing the kubectl invocation with a jsonpath or
custom-columns query (e.g., kubectl get nodes -o jsonpath='{range
.items[*]}{.metadata.name}
{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}' or equivalent),
then in the loop use ssh "$IP" to run sudo journalctl -u kubelet.service --since
'1 hour ago' and redirect to "${NODE}.kubelet.log" so SSH targets the reachable
IP while filenames still use the node name.
- Line 43: Update the paragraph to correct the factual error: state that the
kubeadm systemd unit template expands four variables in ExecStart —
$KUBELET_KUBECONFIG_ARGS, $KUBELET_CONFIG_ARGS, $KUBELET_KUBEADM_ARGS, and
$KUBELET_EXTRA_ARGS — and remove the claim that KUBELET_EXTRA_ARGS is "silently
ignored"; instead note that KUBELET_EXTRA_ARGS (sourced from
/etc/default/kubelet on DEB or /etc/sysconfig/kubelet on RPM) is an officially
supported override and has the highest priority among argument sources, while
still clarifying that setting an unrelated env var like KUBELET_LOG_LEVEL will
not affect kubelet args unless placed into one of those expanded variables.
- Line 35: The sed command in the diff (the expression matching
's/^\s*verbosity:.*/verbosity: 4/') writes a top-level verbosity key, but
kubelet expects logging.verbosity; update the fix to set logging.verbosity
instead of a top-level field. Replace the current sed with logic that updates or
creates the nested logging: block and its verbosity: entry (for example, use yq
to set '.logging.verbosity = 4' or with sed/awk ensure a logging: block exists
and then set/append ' verbosity: 4'); ensure you update the exact command that
currently matches '^\s*verbosity:' so it targets 'logging.verbosity' (or
manipulates the logging: section) rather than adding a top-level verbosity key.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: abb33d3c-de7e-4afe-aec4-d7b997619f94
📒 Files selected for processing (1)
docs/en/solutions/Configure_Kubelet_Log_Level_Verbosity.md
|
|
||
| ```bash | ||
| # On the target node | ||
| sudo sed -i 's/^\s*verbosity:.*/verbosity: 4/; t; $a\verbosity: 4' /var/lib/kubelet/config.yaml |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify actual key path used by kubelet config on a node
sudo grep -nE '^\s*logging:|^\s*verbosity:' /var/lib/kubelet/config.yamlRepository: alauda/knowledge
Length of output: 104
🏁 Script executed:
# Search for kubelet config examples or references in the repository
rg -i 'kubelet.*config|logging\.verbosity|verbosity' docs/ --type md -A 2 -B 2Repository: alauda/knowledge
Length of output: 11838
🏁 Script executed:
# Check if there are any existing kubelet configuration examples in the repo
fd -e yaml -e json -e md | xargs rg -l 'kubelet' | head -20Repository: alauda/knowledge
Length of output: 400
🌐 Web query:
Kubernetes kubelet configuration verbosity field path schema logging.verbosity
💡 Result:
The Kubernetes kubelet configuration includes a logging.verbosity field in the KubeletConfiguration struct (v1beta1 API). This field is nested under the top-level logging object and specifies the log verbosity threshold as an integer (VerbosityLevel, alias of uint32). Default is 0 (only most important messages); higher values (e.g., 2-4 for debug, 5-8 for trace) enable more detailed logging. Error messages are always logged. Full path in YAML/JSON config: logging.verbosity Example from official docs: logging: format: "text" flushFrequency: "5s" verbosity: 3 options: json: infoBufferSize: "0" Schema details: - Type: VerbosityLevel (uint32) - Description: Threshold for log messages; higher values log more details. - Appears in: LoggingConfiguration within KubeletConfiguration. Configure via --config flag pointing to a YAML/JSON file matching this schema. Command-line --v flag is deprecated in favor of config file for this parameter.
Citations:
- 1: https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
- 2: http://k8s.io/docs/reference/config-api/kubelet-config.v1beta1/
- 3: https://v1-33.docs.kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1
- 4: https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/
- 5: https://k8s.io/docs/reference/config-api/kubelet-config.v1beta1/
- 6: https://v1-34.docs.kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/
verbosity field path does not match KubeletConfiguration schema.
Per the official Kubernetes kubelet config schema (v1beta1), verbosity is nested under logging.verbosity, not a top-level verbosity field. The sed command will create a top-level field that kubelet ignores, causing the configuration to silently fail.
Suggested fix
- sudo sed -i 's/^\s*verbosity:.*/verbosity: 4/; t; $a\verbosity: 4' /var/lib/kubelet/config.yaml
+ sudo yq -i '.logging.verbosity = 4' /var/lib/kubelet/config.yaml📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| sudo sed -i 's/^\s*verbosity:.*/verbosity: 4/; t; $a\verbosity: 4' /var/lib/kubelet/config.yaml | |
| sudo yq -i '.logging.verbosity = 4' /var/lib/kubelet/config.yaml |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/en/solutions/Configure_Kubelet_Log_Level_Verbosity.md` at line 35, The
sed command in the diff (the expression matching 's/^\s*verbosity:.*/verbosity:
4/') writes a top-level verbosity key, but kubelet expects logging.verbosity;
update the fix to set logging.verbosity instead of a top-level field. Replace
the current sed with logic that updates or creates the nested logging: block and
its verbosity: entry (for example, use yq to set '.logging.verbosity = 4' or
with sed/awk ensure a logging: block exists and then set/append ' verbosity:
4'); ensure you update the exact command that currently matches '^\s*verbosity:'
so it targets 'logging.verbosity' (or manipulates the logging: section) rather
than adding a top-level verbosity key.
|
|
||
| ### Persistent Configuration — systemd drop-in (fallback) | ||
|
|
||
| If you cannot edit `config.yaml` (some operator-managed setups lock the file), override the kubelet `ExecStart` via a drop-in that **inlines the `--v` flag directly**. Setting a bare environment variable like `KUBELET_LOG_LEVEL=4` does **not** raise verbosity — the stock kubeadm systemd unit only expands the three specific variables `$KUBELET_KUBECONFIG_ARGS`, `$KUBELET_CONFIG_ARGS`, and `$KUBELET_KUBEADM_ARGS`; any other name (including `KUBELET_LOG_LEVEL` or `KUBELET_EXTRA_ARGS`) is silently ignored. |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Inspect real kubelet unit/env expansion on target nodes
systemctl cat kubelet | sed -n '/^\[Service\]/,/^\[/p'
systemctl show kubelet -p FragmentPath -p DropInPaths -p EnvironmentFilesRepository: alauda/knowledge
Length of output: 157
🏁 Script executed:
# Search the repository for references to kubelet systemd units, KUBELET_EXTRA_ARGS, or ExecStart
rg -i "kubelet.*extra.*args|KUBELET_EXTRA_ARGS|ExecStart.*kubelet" --type md --type txt --type yaml --type jsonRepository: alauda/knowledge
Length of output: 1200
🏁 Script executed:
# Look for any kubeadm or kubelet configuration files in the repository
fd -i "kubelet|kubeadm" -type f | head -20Repository: alauda/knowledge
Length of output: 230
🌐 Web query:
kubeadm systemd unit template kubelet environment variables KUBELET_EXTRA_ARGS official documentation
💡 Result:
The official Kubernetes documentation describes the kubeadm systemd unit template for the kubelet in the page "Configuring each kubelet in your cluster using kubeadm" at https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/kubelet-integration/. The key drop-in file is /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf, which augments the base kubelet.service. It sources environment variables including KUBELET_EXTRA_ARGS from /etc/default/kubelet (DEB packages) or /etc/sysconfig/kubelet (RPM packages). Here is the template content from the official docs: [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" # This is a file that "kubeadm init" and "kubeadm join" generate at runtime, populating # the KUBELET_KUBEADM_ARGS variable dynamically EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, # the user should use the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. # KUBELET_EXTRA_ARGS should be sourced from this file. EnvironmentFile=-/etc/default/kubelet ExecStart= ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS To use KUBELET_EXTRA_ARGS, add it to /etc/default/kubelet (or /etc/sysconfig/kubelet on RPM), e.g.: KUBELET_EXTRA_ARGS="--fail-swap-on=false --max-pods=110" Then run systemctl daemon-reload && systemctl restart kubelet. This has highest priority among the arg sources. Prefer NodeRegistration.KubeletExtraArgs in kubeadm config files instead.
Citations:
- 1: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/kubelet-integration/
- 2: http://k8s.io/docs/setup/production-environment/tools/kubeadm/kubelet-integration/
- 3: https://v1-32.docs.kubernetes.io/docs/setup/production-environment/tools/kubeadm/kubelet-integration
- 4: Include the kubelet.service file in the documentation kubernetes/website#29943
Correct the claim about expanded variables in kubeadm systemd units.
The official Kubernetes documentation shows that the stock kubeadm systemd unit template DOES expand $KUBELET_EXTRA_ARGS in the ExecStart line. The template sources this from /etc/default/kubelet (DEB) or /etc/sysconfig/kubelet (RPM). Saying that KUBELET_EXTRA_ARGS is "silently ignored" is factually incorrect — it is an officially supported mechanism with the highest priority among argument sources. The documentation should specify all four expanded variables ($KUBELET_KUBECONFIG_ARGS, $KUBELET_CONFIG_ARGS, $KUBELET_KUBEADM_ARGS, and $KUBELET_EXTRA_ARGS) rather than claiming only three are supported.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/en/solutions/Configure_Kubelet_Log_Level_Verbosity.md` at line 43,
Update the paragraph to correct the factual error: state that the kubeadm
systemd unit template expands four variables in ExecStart —
$KUBELET_KUBECONFIG_ARGS, $KUBELET_CONFIG_ARGS, $KUBELET_KUBEADM_ARGS, and
$KUBELET_EXTRA_ARGS — and remove the claim that KUBELET_EXTRA_ARGS is "silently
ignored"; instead note that KUBELET_EXTRA_ARGS (sourced from
/etc/default/kubelet on DEB or /etc/sysconfig/kubelet on RPM) is an officially
supported override and has the highest priority among argument sources, while
still clarifying that setting an unrelated env var like KUBELET_LOG_LEVEL will
not affect kubelet args unless placed into one of those expanded variables.
| for n in $(kubectl get nodes --no-headers | awk '{print $1}'); do | ||
| ssh "$n" "sudo journalctl -u kubelet.service --since '1 hour ago'" > "${n}.kubelet.log" | ||
| done |
There was a problem hiding this comment.
Using Kubernetes node name as SSH host is brittle.
ssh "$n" often fails because node name is not guaranteed to be DNS-resolvable/reachable from the runner host.
Portable alternative using InternalIP
-for n in $(kubectl get nodes --no-headers | awk '{print $1}'); do
- ssh "$n" "sudo journalctl -u kubelet.service --since '1 hour ago'" > "${n}.kubelet.log"
+kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{range .status.addresses[*]}{.type}={.address}{" "}{end}{"\n"}{end}' \
+| while read -r name addrs; do
+ ip=$(printf '%s\n' "$addrs" | tr ' ' '\n' | awk -F= '$1=="InternalIP"{print $2; exit}')
+ [ -n "$ip" ] && ssh "$ip" "sudo journalctl -u kubelet.service --since '1 hour ago'" > "${name}.kubelet.log"
done📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| for n in $(kubectl get nodes --no-headers | awk '{print $1}'); do | |
| ssh "$n" "sudo journalctl -u kubelet.service --since '1 hour ago'" > "${n}.kubelet.log" | |
| done | |
| kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{range .status.addresses[*]}{.type}={.address}{" "}{end}{"\n"}{end}' \ | |
| | while read -r name addrs; do | |
| ip=$(printf '%s\n' "$addrs" | tr ' ' '\n' | awk -F= '$1=="InternalIP"{print $2; exit}') | |
| [ -n "$ip" ] && ssh "$ip" "sudo journalctl -u kubelet.service --since '1 hour ago'" > "${name}.kubelet.log" | |
| done |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/en/solutions/Configure_Kubelet_Log_Level_Verbosity.md` around lines 126
- 128, The loop uses node names (variable n and ssh "$n") which may not be
DNS-resolvable; change the selection to use each node's InternalIP and keep the
node name for filenames: iterate over pairs (node name and InternalIP) by
replacing the kubectl invocation with a jsonpath or custom-columns query (e.g.,
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}
{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}' or equivalent),
then in the loop use ssh "$IP" to run sudo journalctl -u kubelet.service --since
'1 hour ago' and redirect to "${NODE}.kubelet.log" so SSH targets the reachable
IP while filenames still use the node name.
1 participant
新增一篇 ACP KB 文章，归入
configure区域。✅ 自动化验证通过 — 2 / 2 条验证步骤在真实 Kubernetes 集群上按文章命令跑通（2026-04-22T13:12:03Z）。
configure区域建议 reviewer按
kb/OWNERS.md+kb/KB_REVIEWERS.md该区域的活跃人自动挑选，@ 错了请无视。@changluyi @zhangzujian @oilbeater
没有 GitHub handle 的贡献者（本区域相关请人工 ping）：